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ABSTRACT 

We introduce a protocol which routes on flat, location- 
independent identifiers with guaranteed scalability and 
low stretch. Our design builds on theoretical advances 
in the area of compact routing, and is the first to realize 
these guarantees in a dynamic distributed setting. 

1. INTRODUCTION 

Routing scalability is highly desirable for very large, 
dynamic, and resource-constrained networks, including 
many peer-to-peer systems and the Internet. Shortest- 
path routing algorithms (link state, distance vector, 
path vector, etc.) all [16] require r2(n) memory at each 
router for a network with n destinations, and at least 
as much communication and computation to build the 
routing tables. 

One way to scale routing is to use a structured net- 
work topology that makes routing easy. Examples range 



21[|28 42 . Flat names are a paradigm shift for the net- 



from torus networks in supercomputers 25 to planar 



networks which permit greedy geographic routing 23 



to hypercubes, small world networks, and other topolo- 
gies m distributed hash tables (39||43]|50] . But requirmg 
a particular highly structured topology is not feasible 
for general-purpose networks. 

For general networks, the common scaling technique 
is hierarchy: routing is performed over high-level ag- 
gregate units until it reaches the destination's unit, at 
which point routing proceeds at a finer granularity. For 
example, the Internet routes at the level of IP prefixes 
to a destination domain, and then over an intradomain 
routing protocol to a subnet. Hierarchy has two main 
problems. First, it can have arbitrarily high stretch, 
defined as the ratio of route length to the shortest path 
length. Simultaneously guaranteeing scalability and low 
stretch on arbitrary networks is a nontrivial problem 
which no deployed routing protocols achieve. Second, 
hierarchy requires location-dependent addresses, com- 
plicating management, mobility, and multihoming. 

In response, numerous recent proposals suggest rout- 
ing on location-independent flat names [5| [T2|[T3{[20{ 

This technical report extends our ACM CoNEXT 2010 pa- 
per 41 by including the proofs for the theoretical results. 



work layer: rather than requiring a location-dependent 
IP address to serve the needs of the routing protocol, a 
name is an arbitrary bit string that can serve the needs 
of the application layer. For example, a name could be 
a DNS name, a MAC address, or a secure self-certifying 
identifier 21 28p2 . But no previously proposed proto- 
cols exist for scalable, low-stretch routing on flat names. 

This paper introduces a new routing protocol: Dis- 
tributed Compact Routing, or "Disco" . Disco builds on 
theoretical advances in centralized, static compact rout- 
ing algorithms f3''44' and is the first dynamic distributed 
protocol to guarantee the following properties: 

Scalability: A Disco router needs just 0{y/n) routing 
table entrief[^ regardless of network topology. 

Low stretch: Disco has worst-case stretch 7 on a flow's 
first packet, worst-case stretch 3 on subsequent 
packets, and much lower average stretch. 

Flat names: Disco routes on arbitrary "flat" names 
rather than hierarchical addresses. 

Disco's guarantee of 0{^/n) routing table entries trans- 
lates to 0{r^/n) hits of state where r is the size of a 
partial route; see discussion in J|2j 

Recently, several papers have introduced techniques 
which approach the above properties. In particular, 
citing the lack of a distributed compact routing pro- 
tocol, VRR ^ introduced a novel DHT-inspired ap- 
proach to route on flat names, but as we will see, VRR 



does not guarantee scalability and low stretch. S4 34 



is a distributed implementation of a compact routing 
algorithm of 44 , but it does not route on flat names 



and as we will show, it breaks the state bound of 44 



causing high per-node state on realistic topologies. Dis- 
tributed compact routing was listed as an open problem 
by Gavoille [Tt]. 

We find, however, that guaranteed scalable, efficient 
routing on fiat names is achievable. Disco is a careful 
synthesis of compact routing theory with well-understood 

^The standard notation O(-) hides polylogarithmic factors 
to aid readability. Disco actually has O{\fn\ogn) entries. 
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systems techniques and a novel, low-overhead overlay 
network for disseminating routing state. That synthe- 
sis is the primary contribution of this paper. Disco thus 
represents a step towards closing the gap between the- 
oretical and applied work on scalable routing. 

The rest of this paper proceeds as follows. Sec.[2]dis- 
cusses the need for our three requirements of scalability, 
low stretch, and flat names. We describe how past work 
has fallen short of these requirements in Sec. [3) Sec. |4] 
presents our protocol and Sec. [5] evaluates it. We con- 
clude with a discussion of future work in Sec. [G] 

2. REQUIREMENTS 

This paper is guided by three key requirements. 

Guaranteed scalability: The routing protocol should 
use little space and messaging regardless of the network 
topology. In particular, we wish to reduce state require- 
ments from the Q{n) bits at each node used by tradi- 
tional routing protocols in an n-node network, to some- 
thing asymptotically smaller, e.g., 0[^/n) per node. Guar- 
anteeing scalability on any topology helps ensure that 
the protocol will continue to function smoothly despite 
future growth, higher dynamics, and new environments. 

Disco meets this requirement for graphs in which we 
have a bound on the size of "explicit routes" within 
the local vicinity of each node. In particular. Disco has 
0{y/n) routing table entries for a total of 0{r^Jn) bits of 
state where r is the maximum size of an explicit route, 
since these routes are embedded in nodes' addresses. 
(For example, we might have r — O{\ogn).) In j |4.2[ 
we discuss the need for this assumption and argue that 
it is reasonable. While in the worst case r = O(y^), 
in a router-level Internet map our addresses are shorter 
than IPv6 addresses. 

Guaranteed low stretch: The protocol should find 
paths that are close to the shortest possible. Stretch is 
the ratio of the protocol's route length to the shortest 
path length. Since it is not possible to route on short- 
est paths using o{n) state per node [16 , some stretch 
is unavoidable given our scaling goal. However, we de- 
sire stretch to be bounded by a small constant in the 
worst case, so that local communication stays local and 
performance does not degrade regardless of the traffic 
demands. 

Flat names: The protocol should route on arbitrary 
node names which may have no relationship with a 
node's location. In particular, this service should op- 
erate while preserving the stretch guarantee. 

Flat names are widely recognized as a useful primi- 
tive. The location-independence of flat names aids mo- 
bility and eliminates the management burden of location- 
based address assignment. Numerous proposed redesigns 
of Internet routing use flat names to cleanly separate 



FARA 12 , HIP 21 , and LISP 13 . Flat names can 



also be self-certifying 28 35 46 , where the name is a 



public key or a hash of a public key. This provides se- 
curity without a public key infrastructure to associate 
keys with objects. Self-certifying names have been pro- 



posed for data objects in content-centric networks 28 
40 , for a persistent and contention- free replacement for 
URLs [46j, and for nodes in an accountable Internet 
architecture [sj. 

If low stretch were not a goal, supporting flat names 
would be straightforward. In particular, the routing 
system could resolve names into addresses — using DNS, 
a consistent hashing database over a set of well-known 
nodes 



14 26 34 



or a DHT [15] — and then route over 
addresses. But these solutions violate our stretch re- 
quirement: the resolution step might travel across the 
world even if the destination is next door Even though 
the resolution step may be needed for only the first 
packet of a flow, this latency may dominate for short 
flows; indeed, sub-second delays make a difference to 
users in web services such as Google [Sj. Moreover, 



these solutions lack fate sharing 11 : a failure far from 



the source-destination path can disrupt communication. 
Similarly, an attacker far from the path could disrupt, 
redirect, or eavesdrop on communication. 

Despite its desirability, scalable low-stretch routing 
on flat names has remained elusive. Indeed, satisfying 
all three requirements is an algorithmically challenging 
goal which remained essentially unsolved from the 1989 
introduction of the problem until 2003 (6l, even in 
the static centralized case. And as we shall see in the 
next section, no distributed solution has been developed. 

3. RELATED WORK 

Scalable routing has a long and broad history, be- 
ginning with Kleinrock and Kamoun's 1977 analysis of 
hierarchical routing [27' and branching into many prac- 
tical and theoretical application domains. We limit our 
discussion to routing protocols intended to scale for ar- 
bitrary topologies. 

Scalable routing in theory. Universal compact rout- 
ing schemes, beginning with early work by Peleg and 
Upfal 3^, bound the size of routing tables and the 
worst-case stretch on arbitrary undirected graphs. A 
series of results improved the state and stretch bounds, 
culminating in Thorup and Zwick [44[ who obtained 
stretch 3 and space 0{^/n) per node, along with a family 
of schemes that have stretch k and space 0(fcn^/('^"'"^)) 
for any odd integer fc > 1. This is essentially optimal, 
since stretch < 3 requires state Vt{n) [l8], stretch < 5 re- 



location from identity, including TRIAD 20 ,13 42 , discuss this issue in ijS] 



^Locality-aware DHTs rely on existence of underlying rout- 
ing infrastructure and hence don't solve this problem. We 
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Scheme 



Low 
stretch 



Shortest-path routing 
XL 31* 

Classic hierarchy 
Landmark routing [45^ 





Flat 
names 



/ 



"for graphs where the size of exphcit routes is bounded 

Figure 1: Distributed routing protocols. Scal- 
able indicates the protocol guarantees o(n) state 
per node; low stretch indicates it has 0(1) or 
O(logn) stretch; flat names indicates it routes 
on flat names with low stretch. 

quires state f2(y^), and in general, subject to a conjee- 



44 



ture of Erdos, stretch < k requires state n{n'^^^'^~^^) 

Those optimal results, however, are for the name- 
dependent model, where node names are chosen by 
the routing algorithm to encode location information — 
effectively, they are addresses. In the name-independent 
model, node names can be arbitrary ("flat") names. 
Abraham et al. [s] obtained stretch 3 with space 0{y/ri), 
and Abraham, Gavoille, and Malkhi 2| obtained stretch 
0{k) with space 0{n^^'' log D) where D is the normal- 
ized diameter of the network. Thus, surprisingly, the 
best name-independent schemes nearly match the name- 
dependent ones. 

Unfortunately, this theoretical work on compact rout- 
ing is far from practical for today's Internet. Most crit- 
ically, it assumes centralized routing table construction 
and a static network. 

Scalable routing in practice. Since compact routing 
algorithms are centralized and often relatively complex, 
it has not been clear how to translate them into practi- 
cal distributed protocols; despite several attempts, re- 
sults on the systems side are limited. Fig. [T] summarizes 
some of the protocols most closely related to Disco. 
Landmark routing 



45 



has a similar motivation to 
our work: small space and stretch with a dynamic dis- 
tributed protocol. However, landmark routing does not 
provide guarantees on either space or stretch for general 
topologies, and does not route on flat names. 



SEATTLE 26 provides an Ethernet-like protocol while 

It looks up 



eliminating Ethernet's use of broadcast 
Ethernet addresses in a consistent hashing database run 
on the routers, and routes along shortest paths after the 
first packet. It therefore does not route on flat names 
with low stretch. SEATTLE improves scalability rela- 
tive to Ethernet, but still scales linearly in the number 
of routers n (and resorts to an Internet-like hierarchy 
for large n, which would cause unbounded stretch). 



XL [31] uses heuristics to significantly improve link 
state routing's messaging scalability, but does not im- 
prove the worst case, or routing table size. 

Virtual Ring Routing (VRR) [o] and Routing on Flat 
Labels (ROFL) [lO] route on fiat names by applying 
techniques from DHTs to a physical network rather 
than in an overlay. However, these schemes have un- 
bounded stretch on general topologies, and high stretch 
in practice; e.g., an average stretch of up to 8 in realis- 
tic topologies 10 . We confirm this in ^ and also show 
that VRR can have very high state for some nodes, even 
though it has low state on average. 

Beacon Vector Routing (BVR) [m] represents a node 
address as a vector of distances to beacons and routes 
greedily with these vectors. While it greatly improves 
scalability, BVR can get stuck in local minima caus- 
ing high stretch, and requires a name resolution step 
handled by the landmarks. 



S4 34 adapts one of the Thorup and Zwick (441 com- 



pact routing schemes to a distributed wireless setting. 
However, their adaptation breaks the per-node state 
bound of 44 ; we show in f|5] that S4 can indeed have 
high per-node state. Moreover, it does not route on flat 
names with low stretch. 

Ford [l5j evaluated a distributed version of ^ with 
6(logn) stretch and state, but 15 does not route on 
flat names with low stretch. Also, Disco chooses a dif- 
ferent point in the tradeoff space, with 0(1) stretch but 
0{y/n) state. Both Disco and 15 assume a bound on 
the size of an explicit route (^|. 

Westphal and Kempf [47j applied static compact rout- 
ing algorithms to support mobility, but only with a fixed 
infrastructure and mobile leaf nodes. 

Krioukov et al. 29 [30| applied static compact routing 
algorithms to Internet-like topologies with promising re- 
sults, but did not develop distributed protocols. 

Related techniques in overlay networks. Distributed 
hash tables f39^,'43','50] may appear to satisfy our goals, 
as they route on fiat names often with O(logn) routing 
table entries and within O(logn) overlay hops. How- 
ever, DHTs have unbounded stretch: even a single over- 
lay hop can be arbitrarily longer than the distance to 
the destination! More fundamentally, DHTs are not 
general-purpose routing protocols: they are overlay net- 
works which depend on the availability of a general- 
purpose routing protocol beneath them. 

Tulip f7 is an overlay network that routes with 0{^/n) 
state and round-trip]^ stretch 2 on fiat names. Both 
our work and Tulip share theoretical techniques based 
on [3]. However, like DHTs, Tulip does not solve the 
network routing problem; its stretch guarantee effec- 
tively assumes a stretch-1 routing protocol beneath it. 
One could modify Tulip's data structures to store routes 

^Note this is a different definition of stretch than the more 
common one-way definition that we use in this paper. 
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instead of addresses to achieve routing functionality. 
The problem then is constructing and maintaining this 
routing state in a distributed and dynamic fashion. Lo- 
cality aware DHTs [4 38 also suffer from this problem. 
It is this problem that Disco resolves. 

In summary, to the best of our knowledge, the only 
proposed distributed protocol which is both scalable 



and has low stretch is 15 , and no previously proposed 



practical protocol retains guarantees on scalability and 
stretch while routing on flat names. 

4. DISTRIBUTED COMPACT ROUTING 

This section begins with our assumptions and defini- 



tions 4.1 1. We then present Disco's main components: 
a name- dependent distributed compact routing proto- 



col, NDDisco (^4.2); a name resolution module (§4731); 



and a distributed location database to achieve name- 
independence (^4.4). Finally, we prove Disco's state 



and stretch guarantees (^4.5|. We defer an evaluation 
of messaging overhead to our simulations i^^- 

4.1 Assumptions and definitions 

We assume we are given an undirected connected net- 
work of n nodes with arbitrary structure and link dis- 
tances (i.e., link latencies or costs). We let w ^ w de- 
note a shortest path from v to w (in terms of distance, 
not hopcount), and let d{v, w) denote the length of this 
path. The name of a node is an arbitrary bit string; 
i.e., a flat, location-independent name. We assume each 
node knows its own name and its neighbors' names, but 
nothing else. Like past schemes, our protocol designates 
some nodes as landmarks, whose function will be de- 
scribed later. We let denote the landmark closest to 
a node v. 

An event occurs with high probability (w.h.p.) if it 
has probability > 1 — n~'^ for some constant c > 1. 

We assume nodes can estimate n. While this could be 
done in many ways, we propose the use of synopsis dif- 
fusion '36l. SD requires only extremely lightweight, un- 
structured gossiping of small "synopses" with neighbors 
and produces robust, accurate estimates (e.g., within 
10% on average using 256-byte synopses). 

4.2 Name-dependent compact routing 

Disco begins with a name-dependent distributed com- 
pact routing protocol, NDDisco. This protocol guar- 
antees worst-case stretch 5 on the first packet of a flow, 
worst-case stretch 3 on subsequent packets, and 0{y/n) 
routing table entries per node. But it is name-dependent: 
we assume the source knows the destination's current 
address (defined below), as opposed to just its name. 
NDDisco is based on a centralized algorithm of 44 . In 



this subsection we describe the components of NDDisco, 
and then compare it with S4 
protocol based on Eil. 



Landmarks. A landmark is a node to which all nodes 
know shortest paths. Landmarks will allow us to con- 
struct end-to-end routes of the form s £ ^ t. Each 
of the two segments will be precomputed by the routing 
protocol, and the full route will be close to the shortest 
if £ is close to t. 

Landmarks are selected uniform-randomly by having 
each node decide locally and independently whether to 
become a landmark. Specifically, each node picks a ran- 
dom number p uniform in [0, 1], and decides to become a 
landmark if p < (log n)/n. Thus, the expected num- 
ber of landmarks is n ■ \J (log n) jn — \Jn log n and by a 
Chernoff bound, there will be 9(v'nTogn) w.h.p. 

Since n can change, nodes will dynamically become, 
or cease to be, landmarks. To minimize churn in the set 
of landmarks, a node v only flips its landmark status if 
n has changed by at least a factor 2 since the last time v 
changed its status. This amortizes the cost of landmark 
churn over the cost of a large number (r2(n)) of node 
joins or leaves. 

Vicinities. Landmarks gave us a way for a source s to 
approximately locate a destination i, but if s and t are 
close, it will be a poor approximation relative to the dis- 
tance between them. To solve this problem, each node 
V learns shortest paths to every node in its vicinity 
V{v\. the 0(i/nlogn) nodes closest to v. These sizes 
ensure that each node has a landmark within its vicinity 
w.h.p., which is necessary for the stretch guarantee. 

Learning paths to landmarks and vicinities. Nodes 
learn shortest paths to landmarks and vicinities via a 
single, standard path vector routing protocol. When 
learning paths, a route announcement is accepted into 
d's routing table if and only if the route's destination 
is a landmark or one of the ^{ypnAogn) closest nodes 
currently advertised to v. The entire routing table is 
then exported to u's neighbors. 

With these rules, the protocol will converge so that v 
knows the landmarks and Viy), for a total of Q{\/n log n) 
routing table entries w.h.p. We note however that the 
control plane requires Q{8\Jn logn) state for a node 
with degree <5, because path vector stores the full set of 
route advertisements received from each neighbor. This 
may be acceptable for low-degree graphs or if routers 
with higher degree are more well-provisioned. We can 
reduce control state to 6(1/ logn) by either forgetting 



the unused announcements as in Forgetful Routing 24 



34 , another distributed 



or by preventing them from being sent by having each 
node inform its neighbors of the "radius" of its vicinity. 

Addresses. The address of node v is the identifier 
of its closest landmark paired with the necessary 
information to forward along ^ v. Addresses are 
location-dependent, of course, but they are only used 
internally by the protocol, and are dynamically updated 
to reflect topology changes. 
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But what is "the necessary information to forward 
along £^ ^ w"? In the version of NDDisco evaluated 
here, it is an explicit route consisting of a list of labels, 
one for each hop along the ^ w path. A node's 
address is thus variable length with size dependent on 
the number of hops to its nearest landmark. The reader 
may notice that in the worst case this address size is 
quite large, as much as 0{y/n) bits in a ring network of 
n nodes. This would be too large for a packet header 
and would explode our state bound since we will have 
to store many addresses on some nodes (^4.3 1. Our 



design decision to use explicit routes thus invites some 
explanation. 

Most importantly, the explicit route is in practice ex- 
tremely compact. Each hop at a node of degree d is 
encoded in 0{\ogd) bits following the format of [l9] . 
We measured the size of explicit routes in CAIDA's 



router-level map of the Internet 48 by picking random 



landmarks and encoding shortest paths from each node 
to its closest landmark as a sequence of these O(logd)- 
bit encodings of the node identifiers on the path. The 
maximum size of our addresses is just 10.625 bytes (less 
than an IPv6 address), the 95th percentile is 5 bytes, 
and the mean — the important metric for the per-node 
state bound — is 2.93 bytes (less than an IPv4 address). 

The explicit route could be eliminated. Briefly, an 
address would be fixed at O(logn) bits; each landmark 
£ would dynamically partition this block of addresses 
among its neighbors in proportion to their number of 
descendants, and this would continue recursively down 
the shortest-path tree rooted at £, analogous to a hier- 
archical assignment of IP addresses. Since this would 
complicate the protocol and actually increase the mean 
address size in practice, we chose the simpler explicit 
route design. 

Routing. A source s can now send to a destination t 
as follows. If t is a landmark or t G V{s), then s can 
route along a shortest path to t. Otherwise, it extracts 
it from t's address and routes along s It t. (Recall 
that for NDDisco, unlike Disco, we assume that s knows 
i's address.) This first packet of the flow has worst-case 
stretch 5, a fact which was shown in ^44^ for a centralized 
algorithm, and which applies to our protocol since it is 
essentially a distributed implementation of ^4] . 

Subsequent packets can do better. Upon receipt of 
the message, t determines whether s E V{t). If so, 
t knows the shortest path even though s didn't (note 
that s S V{t) does not imply t £ V{s)). In this case, t 
informs s of the path s ^ i, and all subsequent packets 
follow this path. This is again essentially equivalent to 



the protocol of ^44j which 44 showed guarantees worst- 
case stretch 3. 



Shortcutting heuristics. In S4 34 , if at any point 
the packet passes through a node which knows a direct 
path to t, then the direct path is followed. We refer to 



this as "To-Destination" shortcutting. We implement 
two further optimizations. First, we try both the for- 
ward and reverse routes s — >■ t and t s, and use the 
shorter of these. When combined with To-Destination, 
we call this "No Path Knowledge" shortcutting. 

Second, a more aggressive optimization is for every 
node along the route to inspect the route and see whether 
it knows a shorter path to any of the nodes along the 
route (via its vicinity routes) in either forward or re- 
verse directions, in which case the route is shortened. 
We call this "Up-Down Stream" shortcutting. The lat- 
ter optimization requires listing the global identifiers of 
every node along the path. This can be done on a single 
initial packet. This can also be combined with using the 
reverse route (referred to as "Path Knowledge" then). 
Our simulations will show that this "Path Knowledge" 
optimization significantly reduces stretch, but due to 
the added complexity, we exclude it from the evaluation 
of our core protocol. All results discussed subsequently 
use the "No Path Knowledge" optimization. 



Comparison with S4. Both NDDisco and S4 
are distributed protocols based on 4' 
S4 is based on the algorithm in Sec. 3 of 44 



34 



Specifically, 
where. 



rather than knowing vicinities as described above, each 
node V knows its "cluster": nodes that are closer to v 
than to their closest landmarks. To adapt 44 to a dis- 



tributed setting, S4 selects uniform-random landmarks 



rather than using the multiple-pass algorithm of 44 



Unfortunately, this breaks the per-node state guaran- 
tee; in f|5] we will see that S4's per-node state can be 
quite high. Briefiy, the problem is that some nodes can 
be close to many nodes in the network, exploding their 
cluster size. (This is why 44 needed to select land- 



marks with a more complex algorithm.) 

NDDisco avoids this problem by having each node v 
store its vicinity (as defined above: the 0{y/n) nodes 
closest to v) rather than its cluster. This enforces a 
bound on the number of routing table entries at each 
node for any network, but does have two consequences. 
First, it requires the source to query the destination's 
vicinity in order to guarantee stretch 3, as described 
above; this design is essentially a distributed version of 
the "handshaking-based" scheme of Thorup and Zwick 
(Sec. 4 of [44]). 

Second, this design leads to our use of explicit routes 
in addresses. S4 ensures that for any node v and its 
closest landmark £„, node v will appear in iy^s cluster. 
But in NDDisco, it is possible that v ^ V{£y). Thus, 
we include the explicit route £y v in v's address, so 
packets sent to £v can be forwarded to v. 

S4 and NDDisco do not route on flat names. In the 
rest of this section, we build Disco, which adds routing 
on flat names on top of NDDisco. 

4.3 Name resolution 
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NDDisco {{ 4.2 ) requires a sender to know the desti- 
nation's current address. We can solve this by running 



a consistent hashing 22 database over the (globally- 
known) set of landmarks, similar to 14p4) . Every node 
is aware of its own address {£v,^v ^ v), so it can in- 
sert it into the database, and other nodes can query the 
database to determine v's address. This state is soft: it 
can be updated, for example, every t minutes and timed 
out after 2t +1 minutes. In our simulator, t = 10. We 
will show (^4.5) that this adds 0{-\/n \ogn) entries to 



each landmark, so the state bound is preserved. 

However, this is only a partial solution, because the 
first packet of a flow may have arbitrarily high stretch. 
As discussed in Sec. [2j this impacts latency, reliability, 
and security. We will, however, use this name-resolution 
database as a component of Disco in the next section. 

4.4 Name-independent compact routing 

Disco is comprised of NDDisco (j |4.2p , name reso- 
lution (j |4.3[ ), and a distributed name database. The 
last step ensures constant stretch while routing on flat 
names, and is what we describe in this section. 

Overview. We build a distributed database which 
maps nodes' names to their addresses. The database 
requires the key properties that (1) any necessary query 
can be accomplished with low stretch, (2) the 0{^/n) 
per-node state bound is not violated, and (3) maintain- 
ing the state in the face of network dynamics requires 
little messaging. We adopt the idea of "color" -grouping 
of nodes from [2], but adapted with sloppy grouping 
and routing schemes which are more amenable to a 
distributed setting. We then design a random overlay 
network organized around these groups which can effi- 
ciently distribute flat-name routing state. 

State. We begin with a well-known hash function h{v) 
(e.g., SHA-2) which maps the node name to a roughly 
uniformly-distributed string of Q{\ogn) bits. Node v is 
a member of a "sloppy group" of nodes that have in 
common the first few bits of h{v). Specifically, let G[v) 
be set of nodes w for which the first k :— [log2(\/n/logn)J 
bits of h{w) match those of h{v). Thus, G{v) will con- 
tain 0{yjn logn) nodes w.h.p. The group is sloppy 
because this definition depends on w's estimate of n, 
which will vary slightly across nodes. Node v then en- 
sures that every node in G{v) stores w's address. We 
will return to how exactly this is done shortly. 

This definition of the sloppy group G{v) has two im- 
portant properties. First, it is consistent in the sense 



of consistent hashing 22 : a small change in the num- 



ber of nodes does not result in a large change in the 
grouping. This is essential to scalably handle dynam- 
ics. More precisely, there will be no change in k and 
hence no change in the grouping unless n changes by a 



constant fact or 

The second important property is that if the group- 
ing does change, it corresponds to splitting a group in 
half or merging two groups. This means that nodes 
that have slightly different opinions about the value of 
n will still roughly agree on the grouping, which helps 
us handle (constant-factor) error in estimating n. 

We show how to maintain the sloppy group state 
shortly; but first, we describe how to use it. 

Routing. To route from s to t, node s first checks (as 
in NDDisco) whether it knows a direct path to t, either 
because i is a landmark or i e V{s). If so, it routes 
directly. 

Otherwise, if s knows Vs address (i.e., because s G 
G[t)), it can route to t according to NDDisco. 

Otherwise, s locally computes h{t). It then examines 
its vicinity and finds the node w G V{s) which has the 
longest prefix match between h{w) and h{t). (This 
can be optimized slightly to be the closest node w with 
a "long enough" prefix match.) Due to our choice of 
k, with high probability, w G G{t), so w will know Vs 
current address [ItJ-t ^ t)- The full path (if no short- 
cutting occurs) is thus s ^ w-^ it t. We show that 
this has stretch < 7 in §4.5| 

After the first packet, s knows i's address, so routing 
proceeds as in NDDisco with stretch < 3. 

Note that there is a vanishingly small but nonzero 
probability that t's address is not found because our 
state is maintained correctly only with high probability. 
Such events never occurred in our simulations. How- 
ever, if this did occur routing could operate correctly by 
simply using name resolution on the landmark database 



{{ 4.3 1 as a fallback. 



Sloppy group maintenance. We now describe how a 
node V can ensure that the nodes G{v) have v's address. 
(Note that v does not know which nodes these are.) 

A naive solution is to use the consistent hashing- 
based name resolution database running on the land- 



marks (^.2|. Each node v already stores its name and 
address at the landmark which owns the key h{v). It 
is easy to see that all of G{v) will be stored on a pre- 
dictable set of O(logn) landmarks, from which any node 
can therefore download its group membership informa- 
tion. This, however, imposes a high messaging burden 
on the landmarks: if every node changes its address 
once per minute, the landmark would have to relay 
0{^yn) addresses to each of 0(\/n) nodes for a total 
of 0{n) bytes per minute. 

Disco instead adopts a more decentralized solution. 
Each node v maintains a set of overlay neighbors 
N{v). Similar to a DHT structure, N{v) includes v's 

''if n is such that k is near the boundary of two values, we can 
avoid the frequent "flapping" that could result by changing 
the sloppy group only when the estimate of n changes by at 
least some constant factor (e.g., 10%). 



6 



successor and predecessor in the circular ordering of 
nodes according to their hash values h{-). N{v) also 
includes a small number of long-distance links called 
"fingers" . 

To select a finger, a node v picks a random hash- value 
a from the part of hash-space that falls within G{v). 
Following [32], a is picked such that the likelihood of 
picking a value is inversely proportional to its distance 
in hash-space from h{v). Based on these rules, a node 
finds the name and address of a finger by querying the 
landmark-based resolution database for the node with 
the closest hash-value to a. In this manner, node v 
picks a small constant number of fingers (we will test 
1 and 3 in our simulations) refreshing the set N{v) pe- 
riodically at a low rate. It then opens and maintains 
TCP connections to each of these nodes, for an average 
of |iV(i))| 4 or 8 overlay connections (for 1 or 3 fin- 
gers respectively) counting both outgoing and incoming 
connections. 

Within this overlay, we can efficiently disseminate 
routing state in a manner very close to a distance vec- 
tor (DV) routing protocol. We describe the protocol 
through its differences from the standard DV proto- 
col. First, we emphasize that we use this protocol only 
to propagate address information, rather than to find 
routes. Second, since we are only interested in dis- 
seminating addresses rather than finding short paths, 
we need not include a distance in the announcement, 
but we (obviously) must include the originating node's 
name and address. Third, nodes only propagate ad- 
vertisements to and from nodes they believe belong to 
their own group, thus keeping address information lo- 
cal to each group. Fourth and most importantly, node v 
propagates advertisements only to those nodes in N{v)r\ 
G{v) which would cause the message to continue in the 
same direction: that is, announcements received from 
an overlay neighbor with higher hash-value are prop- 
agated only to neighbors with lower hash-values, and 
vice-versa. This eliminates distance vector's count-to- 
infinity problem because as announcements are prop- 
agated, their distance (in hash space) from the origin 
of the announcement strictly increases. Other aspects 
of the protocol (incremental updates, state maintained, 
etc.) are similar to DV. 

Why does this design work? First note that although 
nodes differ in their opinion on the sloppy grouping, 
they don't differ substantially. In particular, since we 
can ensure that estimates of n are within a factor of 2 of 
the correct value w.h.p., nodes will differ by at most one 
bit in the number of bits k that they match to determine 
the grouping. Thus, there will be a "core group" G'{v) 
such that all nodes in G'{v) will agree that they are in 
the same group. G'{v) is clearly connected via succes- 
sor/predecessor links, so v's announcement will reach 
all the nodes in G'{v). Since |G"(u)| = Q{^/n logn) a 



Chernoff bound can be used to show that every node 
s will have some member of G' {v) in its vicinity w.h.p. 
The routing protocol above will then find such a node 
with its prefix-matching step, so s will be able to route 
to V. 

Beyond correctness, the design is efficient in two ways. 
First, the overlay has constant average degree, so each 
node will receive only a few copies of each announce- 
ment message. Second, these messages are propagated 
relatively quickly, as the expected path length in the 
Symphony topology is 0(log^ 32 1. 

We briefly clarify two minor points. First, there might 
be conflicting announcements received by v for some 
node x's address; in this case, v can simply use the an- 
nouncement which was received from the node whose 
hash- value is closest to h{x). Second, during conver- 
gence and maintenance of the overlay, v may have all 
its announcements for some node x temporarily with- 
drawn even though x is still live. To provide reliable 
service during these periods, v can delay removal of ad- 
dress state until some short period (e.g., 30 sec) has 
passed. 

4.5 Guarantees 

We next prove that Disco maintains low stretch and 
state. We do not bound the messaging overhead ana- 
lytically, but we will simulate it in fs) 



Stretch. As previously noted, the results of 44 apply 



to our name dependent protocol NDDisco: it has stretch 
< 5 for the first packet of a flow, and < 3 for subsequent 
packets. 

Our name-independent routing, however, lengthens 
paths further. We now show that it still maintains 
stretch < 7 for the first packet. 



Theorem 1. After converging, Disco routes the first 
packet of each flow with stretch < 7, and subsequent 
packets with stretch 3 w.h.p. 

Proof. Packets after the first have stretch 3 as shown 
in 1 44 . For the first packet, there are several special 
cases that can result in lower stretch — if the source s 
knows t's address, t S V{s), t is a landmark, or short- 
cutting occurs; these cases are easy to deal with and we 
omit a discussion. There is also a case that no appro- 
priate w is found and stretch might be higher than 7; 
w.h.p., this does not occur. In the general case, the full 
path is s w ^ t, where w € V{s), it is the 

landmark closest to t, and each segment is a shortest 



^ Since we are gossiping on many paths rather than routing 
along a greedy path, we conjecture tha t th is bound can be 



improved to O(logn) along the lines of 33 
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path. We first show a Useful Fact: 

d{£t,t) ~ d{t,£t) (graph is undirected) 

< d{t,£s) (since £t is t's closest landmark) 

< d{t, s) + d{s, is) (triangle inequality) 
= rf(s, t) + d{s, £s) (graph is undirected) 

< d{s, t) + d(s, t) {is is in V{s) and t isn't) 
= 2d{s,t). 

Note the next-to-last step {ig e holds w.h.p. over 

the random choice of landmarks. 

We can now upper-bound the length of each segment 
of the full path. For the first segment s ^ w, we have 
d{s,w) < d{s,t) since w is within s's vicinity and t is 
not. For the second segment w it, we have 

d{w, it) < d{w, s) + d{s, t) -f d{t, it) (triangle ineq.) 

< d{s,t) +d{s,t) +d{t,it) 

< d{s, t) + d(s, t) + 2d{s, t) (Useful Fact) 

= 4d(s,i)- 

For the third segment it ^ i, we have d{it,t) < 2d{s, t), 
again by the Useful Fact. Adding up the three segments, 
we have that the length of the route is < 7d{s, t). □ 

State. For convenience, we analyze state in terms of 
the number of entries in the protocol's routing tables. 
Each entry may contain a node name and address, which 
in turn contains a landmark name and an explicit route 



from the landmark to the destination. (Recall from { 4.1 



that the explicit route will in practice occupy just a few 
bytes, and for extreme cases a variant of our design can 
ensure 0(logn)-size addresses for any topology.) 

Theorem 2. After converging, with high probability, 
each Disco node of degree S maintains 0{S\/n logn) en- 
tries in its routing tables including the control and data 
planes, or 0{^/nlogn) using forgetful routing. 

Proof. The protocol maintains several kinds of state. 
First, each node of degree S stores path vector routing 
state for the landmarks. Since each node becomes a 
landmark independently with probability -y/ (\ogn)/n, 
there are Q{\/n logn) landmarks in expectation and, by 
a ChernofF bound, with high probability (w.h.p.). Each 
of a node's S neighbors sends it 0{\/n\ogn) landmark 
route announcements, for a total of 0{S\/n logn) en- 



tries, or 0{y/n logn) with forgetful routing (^4.2). Note 



that even without forgetful routing there are 0{^/n logn) 
entries in the data plane. 

Similarly, each node picks the closest &{y/n log n) 
nodes for its vicinity again via path vector, for a total 
of 0{dy/n logn) entries, or 0{y/n log n) with forgetful 
routing. 

To enable compact source routes, each node stores a 
mapping from a compact forwarding label to an outgo- 
ing interface. In general, this will require one entry for 



each of S neighbors. Although it would be unrealistic 
in real topologies, we might have S > ^fn. However, 
the node really needs to remember the mapping only 
for those forwarding labels that will actually be used; 
these will be for the neighbors leading along shortest 
paths to landmarks or nodes in the node's vicinity, of 
which there are at most 0{\Jn logn). 



Landmarks store the name resolution database {\ 4.3 ) 
which has one entry for each of the n nodes. These 
are hashed onto the landmarks according to consistent 
hashing, which in its simplest form with a single hash 
function gives each landmark a factor 0(logn) more 
than their fair share of the keyspace w.h.p. Thus, the 
most overloaded nodes will receive n/Q{\/n logn/ log n) - 
0{\/n logn) name resolution entries in expectation, and 
(by a Chernoff bound) w.h.p. By simply using multiple 
hash functions we can reduce consistent bashing's load 
and end up with 0{y 



imbalance 
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'n/logn) entries. 
The distributed name database (j ]4.4[ ) has each node 
store O(logn) overlay neighbors, and the names and 
addresses of nodes in its sloppy group. Recall that the 
group for node v is defined as those nodes which share 
the first [log2(v/n7lo&ri) + 0(1)J bits of h{v). This 
includes Q{y/n logn) nodes in expectation and, again 
by a Chernoff bound, with high probability. 

Thus, in total the algorithm requires 0{\/n\ogn) rout- 
ing table entries. □ 

5. EVALUATION 

We evaluate Disco in comparison with two recent pro- 
posals for scalable routing, S4 and VRR, over a variety 
of networks. Our results confirm our main goals. First, 
we find Disco has an extremely balanced distribution 
of state across nodes, and across topologies, while in 
both S4 and VRR some nodes have a very large amount 
of routing state in realistic topologies. Second, Disco 
maintains low stretch in all cases, even for the first 
packet of a fiow, while the other protocols can have 
high first-packet stretch; this is particularly evident for 
a topology annotated with link latencies rather than 
simply hopcount. 

5.1 Methodology 

Protocols. We evaluate five protocols: Disco, ND- 
Disco, S4 [34], VRR [9], and path vector routing. ND- 
Disco is our name-dependent protocol coupled with the 
landmark-based name resolution database; it is directly 
comparable to S4 in its goals but guarantees 0{^/n) 
per-node state. Our implementation of S4 is as in [34| 
except that we use path vector for cluster and landmark 
routing, making it more comparable to NDDisco. We 
evaluated VRR with r — 4 virtual neighbors as in [9]. 
VRR's converged state depends on the order of node 
joins; we start with a random node and grow the con- 
nected component of joined nodes outward. 
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Simulators. We simulated Disco, S4, and path vec- 
tor in a custom discrete event simulator. For topolo- 
gies larger than 1024 nodes, we built a static simulator 
which calculates the post-convergence state of the net- 
work. We evaluate VRR in the static simulator only. 
Our simulator for VRR scales poorly so we present only 
results on 1024 node topologies for VRR. 

In many cases, for large topologies, we sample a frac- 
tion of nodes or source-destination pairs to compute 
state, stretch, and congestion. 

Topologies. Our results include (1) a 30,610-node 
AS-level map of the Internet [49]; (2) a 192,244-node 



router-level map of the Internet [48|; (3) G(n,m) ran- 
dom graphs of various sizes, i.e., n nodes with m uniform- 
random edges, with m set so that the average degree is 
8; and (4) geometric random graphs of various sizes with 
average degree 8. 

5.2 Results 

State. We measure data plane state for the protocols. 
This includes everything necessary to forward a packet 
after the protocol has converged: forwarding entries for 
landmarks and vicinities, name resolution entries on the 
landmark database, forwarding label mappings for our 
compact source route format in NDDisco, and the ad- 
dress mappings for Disco. 

Fig. [2] shows S4 does well on the random graphs, 
but is extremely unbalanced on the Internet topologies. 
Intuitively, S4 does poorly on topologies where some 
nodes are more "central" than others. In fact, it is easy 
to show that S4 will have 0(n) state on some nodes in 
the worst case|^ This demonstrates that S4's simplifi- 
cation of one of the algorithms of 44 can indeed cause 
high state. 

In contrast. Disco and NDDisco have very balanced 
distributions of state in all cases. Note that NDDisco 
is a fairer comparison with S4 since both protocols are 
name-dependent, while Disco adds additional state for 
name-independence. Average state is slightly higher 
in NDDisco than S4 because of a differing design de- 
cision about vicinity size: S4 expands its cluster until 
it reaches a landmark, while NDDisco and Disco have 
vicinities which are fixed at Q{y/n logn) nodes. This 

^Consider a tree whose root has y'n children at distance 
1; each of these children has ^/n children along an edge 
of distance 2. S4's version of vicinities is called a cluster. 
each node v knows the nodes w which are closer to v than 
the distance from w to its closest landmark. Consider any 
"grandchild" node v which is not picked as a landmark. The 
distance to its parent is 2, but for most such nodes the parent 
is not a landmark. The distance to its grandparent (the 
root) is 3, but with probability 1 — 0(l/v'n) the root is not a 
landmark. Therefore the closest landmark must be another 
grandchild at distance 4 from v. Thus, the majority of the 
grandchildren nodes must be in the root's cluster, so the 
root requires Q{n) routing table entries. 



difference is not fundamental to NDDisco, but is neces- 
sary for Disco so that there will be an intersection be- 
tween each node's vicinity and each destination's sloppy 
group. 

As Fig. |4] (left) and Fig. [5] (left) show, VRR fares very 
poorly compared to both S4 and Disco on both topolo- 
gies in terms of state. It does even worse than path 
vector for a few nodes. This is because VRR constructs 
end-to-end paths and stores state at each node along 
the path, so in theory it could have as many as 6(n^) 
routing entries at a node (though it does not approach 
this worst case). 

All results on routing state discussed so far only count 
the number of entries. In Table. [Tj we present numbers 
for state in terms of kilobytes of memory. The size of 
source routes is determined using the scheme described 
in |4.2[ As the table shows, the conclusions are similar 
when measuring bytes instead of entries. 

Stretch. Fig. |3] shows the distribution of stretch in 
S4, Disco, and NDDisco. We call the reader's attention 
to the difference between the three graphs. In the In- 
ternet router and AS topologies, links are unweighted 
(or equivalently, all link latencies are 1). Thus, maxi- 
mum stretch is limited simply because the ratio of the 
longest to the shortest path is bounded. In contrast, 
the geometric random graph includes link latencies and 
SA experiences worst-case stretch of 72 while Disco's 
highest stretch is just over 2. Unfortunately, a latency- 
annotated Internet topology was not available to us; it 
is likely that this would qualitatively change the proto- 
cols' stretch. 

After the first packet. Disco does slightly better than 
S4 on random and geometric graphs, both perform sim- 
ilarly on the router-level topology, and S4 does better 
on the AS-level graph. 

Fig. |4] (middle) and Fig. [5] (middle) compare stretch in 
Disco, S4 and VRR over a G{n,m) random graph and 
a geometric random graph with latency values. Like 
state, VRR provides no bounds on stretch. The max- 
imum stretch values seen for the first packets in the 
geometric random graph are 2.4 for Disco, 30 for S4, 
and 39 for VRR. 

As described in §4.2[ the above results use the "No 
Path Knowledge" shortcutting heuristic. Fig. [6] shows 
the relative effect of different shortcutting heuristics. 
Using Path Knowledge, the stretch can be brought very 
close to 1 (last row of table). 

Control messaging. We compare messaging costs 
during initial convergence only, leaving continuous churn 
to future work. The results are shown in Fig. [8j ND- 
Disco's messaging overhead is slightly greater than S4's, 
reflecting its somewhat larger vicinities as discussed above. 
However, Disco has only a small amount of additional 
messaging to support routing on flat names with low 
stretch, demonstrating the efficiency of our dissemina- 
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Figure 2: State in a 16,384-node Geometric Random Graph (left), Internet AS-level graph (middle) 
and Internet Router- level graph (right). 
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Figure 7: Comparison of state at a node 
for the Router-level Internet topology. S4 
does better on average, but severely breaks 
worst-case bounds. 
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Figure 8: Mean messages per node sent un- 
til convergence in path vector, S4, NDDisco and 
Disco (with 1 and 3 fingers for address dissemi- 
nation) for G{n, m) graphs of increasing size. The 
curve for path vector has been linearly extrapo- 
lated beyond 512 nodes. 

tion protocol. We show results for 1 and 3 outgoing 
fingers per node in the address dissemination overlay 
network. The use of a larger number of fingers leads to 
lower overlay diameter, and hence faster convergence, 
at the cost of increased messaging. For instance, for 
a 1024-node G(n, m) topology, with each node picking 
1 outgoing finger, the average and maximum distances 
traveled by address announcements were measured to 
be 5.77 and 24 respectively, while picking 3 random fin- 
gers reduced these numbers to 3.04 and 16. At the same 
time, the number of messages increased by 3.3%. 

Congestion. To compute congestion, we have each 
node route to a random destination and count the num- 
ber of times each edge is used. One might have guessed 
that the compact routing schemes would have high con- 
gestion since many routes pass near landmarks. How- 
ever, Fig. [4] (right) and Fig. |5] (right) show that in syn- 
thetic topologies, congestion is surprisingly close to that 
of shortest-path routing. VRR shows higher congestion 
than all the three other schemes. On the AS-level In- 
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ternet topology (Fig. 10), Disco does experience more 
congestion for a few edges than shortest-path routing. 

Scaling. Fig. |9] shows how Disco, NDDisco and S4 
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Figure 10: On the AS-level Internet topology, a 
small fraction (0.05%) of edges face significantly 
more congestion than shortest-path routing. 

scale with increasing number of nodes n in geometric 
random graphs, showing mean stretch and mean state. 
S4's first-packet stretch remains high, but for the rest 
of the curves, the stretch is similarly low and close to 
1. Routing state grows as 0{y/n). 

Accuracy of static simulation. We used a static 
simulation of the network's state after convergence to 
calculate state, stretch and congestion results for large 
topologies. Our comparison of results from both the 
static simulator and the full discrete event simulator 
shows that the static simulator achieves good accuracy. 
For instance, for the 1024-node random graph, the dif- 
ference between mean stretch as measured by the static 
simulator is within 0.9% for Disco's later packets and 
0.7% for S4's later packets. Both stretches are inflated 
by similarly small amounts. 

Error in Estimating Number of Nodes. The previ- 
ous results assume all nodes know the value of n. Here, 
we inject random errors of up to 60% in this estimation. 
With 60% random error, across 5 runs on the 1024-node 
random graph, only one node failed to find in its vicin- 
ity a node in only one of the sloppy groups, and hence 
failed to reach all destinations in that group. With 40% 
random error, all nodes were able to reach all nodes and 
mean stretch increased marginally by 0.6% from 1.253 
to 1.261. Note that this is an extreme case since we can 
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Figure 9: Disco vs. S4: 
mean stretch (left) and 
mean state (right) in ge- 
ometric random graphs of 
increasing size. 
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6. CONCLUSION 

Traditionally, hierarchy has been the only way to 
scale general-purpose routing. Hierarchy has led to in- 
efficiency in routes, and the use of location-dependent 
addresses which complicate mobility and management. 
This paper stands in a long line of work which has pro- 
gressively brought compact routing, an algorithmically 
promising approach which eschews hierarchy, closer to 
practical reality. Disco takes another step forward by 
providing distributed and dynamic routing on flat names 
with guaranteed scalability and low stretch. 

One area of future work deals with improving our 
stretch bounds and offering different tradeoffs. In par- 
ticular, is it possible to reduce the worst-case first-packet 
stretch from 7 to the optimal 3 in a distributed way? 
Disco has chosen one point in the state/stretch tradeoff 
space, with 0(^/?i) state and stretch < 3 for packets 
after the first; can we translate other tradeoff points to 
a distributed setting for name-independent routing? 

Another significant outstanding question is to what 
extent Disco can support policy routing in the Internet. 
In many ways. Disco can provide a significant amount of 
flexibility. For example, although Disco chooses land- 
marks randomly, its state and stretch stretch bounds 
require only that each node has at least one landmark 
within its vicinity and that there are Oi^^n) total land- 
marks. These rules would permit an operator to choose 
landmarks in non-random ways, for example to pick a 
more well-provisioned landmark, ensure that a node's 
landmark is within its own domain, or use a landmark 
service supplied by a network provider. And to main- 
tain a globally scalable infrastructure with 0{ypri) to- 
tal landmarks, landmark identifiers could be purchased 
from or allocated by a registry, much as AS numbers are 
today. However, policies also pose challenges. Disco 
assumes that the route v 1^ can also be used in 
the reverse direction in w's address in order to route 
£y V, and assumes similar reversibility for vicinity 
routes (when checking the destination's vicinity for a 
path from the source), which would limit the possi- 
ble policies. A second problem is that policy rout- 



ing can significantly lengthen paths; routing through 
the landmark nearest to a destination many not pro- 
vide a stretch guarantee for general policies, even when 
the route length is compared to the shortest policy- 
compliant path. Resolving these challenges would be 
an interesting area of future work. 

We thank our shepherd, Bruce Maggs, and the anony- 
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